AutoBERT-Zero: Evolving BERT Backbone from Scratch

Transformer-based pre-trained language models like BERT and its variants have recently achieved promising performance in various natural processing (NLP) tasks. However, the conventional paradigm constructs backbone by purely stacking manually designed global self-attention layers, introducing inductive bias thus leads to sub-optimal. In this work, we make first attempt automatically discover novel model (PLM) on a flexible search space containing most fundamental operations from scratch. Specifically, propose well-designed which (i) contains primitive math intra-layer level explore attention structures, (ii) leverages convolution blocks be supplementary for attentions inter-layer better learn local dependency. To enhance efficiency finding architectures, an Operation-Priority Neural Architecture Search (OP-NAS) algorithm, optimizes both algorithm evaluation of candidate models. (OP) evolution strategy facilitate via balancing exploration exploitation. Furthermore, design Bi-branch Weight-Sharing (BIWS) training fast evaluation. Extensive experiments show that searched architecture (named AutoBERT-Zero) significantly outperforms different capacities downstream tasks, proving architecture's transfer scaling abilities. Remarkably, AutoBERT-Zero-base RoBERTa-base (using much more data) BERT-large (with larger size) 2.4 1.4 higher score GLUE test set.